I ran the experiments on ami sdm1. With the bug fixed, the "fast" version of tdnn_lstm+chain is slightly worse than the non-fast one, while the "fast" version of blstm+chain is slightly better than its non-fast counterpart.
dev eval
fast tdnn_lstm 38.0 41.3
non-fast tdnn_lstm 37.8 40.8
fast blstm 39.5 42.9
non-fast blstm 39.7 43.3

The fast and non-fast might be still not totally comparable because of the way the stats are zeroed as mentioned by @danpovey in the issue #1312

fast version before bug fix:

fast tdnn_lstm 37.7 40.6
fast blstm 39.7 42.9

tdnn_lstm before/after fix:
exp/sdm1/chain_cleaned/tdnn_lstm1i_adversarial0.0_fast_sp_bi_ihmali_ld5: num-iters=87 nj=2..12 num-params=43.4M dim=40+100->3732 combine=-0.16->-0.15 xent:train/valid[57,86,final]=(-1.83,-1.54,-1.52/-2.18,-2.11,-2.11) logprob:train/valid[57,86,final]=(-0.170,-0.129,-0.125/-0.239,-0.243,-0.242)
exp/sdm1/chain_cleaned/tdnn_lstm1i_bug_test_sp_bi_ihmali_ld5: num-iters=87 nj=2..12 num-params=43.4M dim=40+100->3732 combine=-0.16->-0.15 xent:train/valid[57,86,final]=(-1.83,-1.51,-1.51/-2.21,-2.14,-2.13) logprob:train/valid[57,86,final]=(-0.170,-0.127,-0.123/-0.245,-0.252,-0.248)

blstm before/after fix:
exp/sdm1/chain_cleaned/blstm_1a_adversarial0.0_fast_sp_bi_ihmali: num-iters=109 nj=2..12 num-params=11.6M dim=40+100->3732 combine=-0.18->-0.17 xent:train/valid[71,108,final]=(-1.90,-1.67,-1.67/-2.18,-2.12,-2.12) logprob:train/valid[71,108,final]=(-0.180,-0.145,-0.143/-0.242,-0.243,-0.241)
exp/sdm1/chain_cleaned/blstm_1a_fast_bug_test_sp_bi_ihmali: num-iters=109 nj=2..12 num-params=11.6M dim=40+100->3732 combine=-0.18->-0.17 xent:train/valid[71,108,final]=(-1.92,-1.67,-1.68/-2.20,-2.12,-2.11) logprob:train/valid[71,108,final]=(-0.179,-0.142,-0.140/-0.240,-0.239,-0.237)

The path is:
/export/a12/ywang/kaldi/egs/ami/s5b/

I couldn't find any obvious reason why the WER would be worse [and the optimization was better].. I'm pretty sure this is just random noise we're seeing. And it's a clear bug-fix. So I'm merging.